Parkinson's Disease Prediction using Ensemble Techniques¶
- Submitted by: Kovendhan V
- PGP.AIML.O.JUN20B.Group4
Title: Parkinsons Disease Data Set
Abstract: Oxford Parkinson's Disease Detection Dataset
Source:
The dataset was created by Max Little of the University of Oxford, in collaboration with the National Centre for Voice and Speech, Denver, Colorado, who recorded the speech signals. The original study published the feature extraction methods for general voice disorders.
Data Description & Context:
Parkinson’s Disease (PD) is a degenerative neurological disorder marked by decreased dopamine levels in the brain. It manifests itself through a deterioration of movement, including the presence of tremors and stiffness. There is commonly a marked effect on speech, including dysarthria (difficulty articulating sounds), hypophonia (lowered volume), and monotone (reduced pitch range). Additionally, cognitive impairments and changes in mood can occur, and risk of dementia is increased.
Traditional diagnosis of Parkinson’s Disease involves a clinician taking a neurological history of the patient and observing motor skills in various situations. Since there is no definitive laboratory test to diagnose PD, diagnosis is often difficult, particularly in the early stages when motor effects are not yet severe. Monitoring progression of the disease over time requires repeated clinic visits by the patient. An effective screening process, particularly one that doesn’t require a clinic visit, would be beneficial. Since PD patients exhibit characteristic vocal features, voice recordings are a useful and non-invasive tool for diagnosis. If machine learning algorithms could be applied to a voice recording dataset to accurately diagnosis PD, this would be an effective screening step prior to an appointment with a clinician
Attribute Information:
name- ASCII subject name and recording numberMDVP:Fo(Hz)- Average vocal fundamental frequencyMDVP:Fhi(Hz)- Maximum vocal fundamental frequencyMDVP:Flo(Hz)- Minimum vocal fundamental frequencyMDVP:Jitter(%),MDVP:Jitter(Abs),MDVP:RAP,MDVP:PPQ,Jitter:DDP- Several measures of variation in fundamental frequencyMDVP:Shimmer,MDVP:Shimmer(dB),Shimmer:APQ3,Shimmer:APQ5,MDVP:APQ,Shimmer:DDA- Several measures of variation in amplitudeNHR,HNR- Two measures of ratio of noise to tonal components in the voicestatus- Health status of the subject (one) - Parkinson's, (zero) - healthyRPDE,D2- Two nonlinear dynamical complexity measuresDFA- Signal fractal scaling exponentspread1,spread2,PPE- Three nonlinear measures of fundamental - frequency variation
Objective:
Goal is to classify the patients into the respective labels using the attributes from their voice recordings.
Steps and tasks:
- Load the dataset
- It is always a good practice to eye-ball raw data to get a feel of the data in terms of number of records, structure of the file, number of attributes, types of attributes and a general idea of likely challenges in the dataset. Mention a few comments in this regard (5 points)
- Using univariate & bivariate analysis to check the individual attributes for their basic statistics such as central values, spread, tails, relationships between variables etc. mention your observations (15 points)
- Split the dataset into training and test set in the ratio of 70:30 (Training:Test) (5 points)
- Prepare the data for training - Scale the data if necessary, get rid of missing values (if any) etc (5 points)
- Train at least 3 standard classification algorithms - Logistic Regression, Naive Bayes’, SVM, k-NN etc, and note down their accuracies on the test data (10 points)
- Train a meta-classifier and note the accuracy on test data (10 points)
- Train at least one standard Ensemble model - Random forest, Bagging, Boosting etc, and note the accuracy (10 points)
- Compare all the models (minimum 5) and pick the best one among them (10 points)
Step 1: Import the necessary Libraries¶
In [1]:
#Numerical Calculations
import numpy as np
import pandas as pd
from scipy.stats import norm, shapiro
#Data Visualization
import matplotlib.pyplot as plt
plt.style.use('ggplot')
import seaborn as sns
from statsmodels.stats.outliers_influence import variance_inflation_factor as vif
from sklearn import tree
from sklearn.tree import export_graphviz
from IPython.display import Image
#Data Preprocessing
from sklearn.preprocessing import RobustScaler
#Train and Test preparation
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV, StratifiedKFold
#Model Building - Classifiers
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
#Model Building - Ensembles
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, StackingClassifier
import xgboost as xgb
from lightgbm import LGBMClassifier, plot_importance
#Model Evaluation
from sklearn.metrics import confusion_matrix, classification_report, roc_curve
from sklearn.metrics import accuracy_score, roc_auc_score, f1_score, precision_score, recall_score, log_loss
from sklearn.metrics import plot_roc_curve, plot_confusion_matrix, plot_precision_recall_curve
from sklearn.inspection import plot_partial_dependence
#Misc
import os
import gc
import warnings
warnings.filterwarnings('ignore')
Step 2: Load the dataset¶
Loading the dataset and checking the first 10 records
In [2]:
parkinsons = pd.read_csv("/content/drive/My Drive/Colab Notebooks/Datasets/Data - Parkinsons")
parkinsons.head(10)
Out[2]:
Checking the Features / Variables
In [3]:
parkinsons.columns
Out[3]:
Checking the shape of the dataset
In [4]:
print('No of Rows : ', parkinsons.shape[0])
print('No of Columns : ', parkinsons.shape[1])
Checking the datatypes and null records
In [5]:
parkinsons.info()
Checking total number of Null values in the dataset
In [6]:
parkinsons.isnull().any().sum()
Out[6]:
Observations:
- For each patient total of six records available in the dataset. This might cause 'Sample Bias' in the model.
- There are no null records.
- However let us check if the values are all numeric.
In [7]:
#Checking if the dataset has only Numeric Data
parkinsons.applymap(np.isreal).all()
Out[7]:
Re-ordering the target variable 'status' as the last column for our convenience
In [8]:
temp = parkinsons['status']
parkinsons.drop(['status'], axis = 1, inplace=True)
parkinsons['status'] = temp
parkinsons.head()
Out[8]:
Step 3: Statistical Summary (Five Number Summary) of the Dataset¶
In [9]:
parkinsons.describe().T
Out[9]:
Observations from the Statistical Summary:
Scale of all the variables are different, hence it becomes imperative to use Scaler transformation before Model Building.
MDVP:Fo(Hz):- Min-Max Range of 171.77 and Standard Deviation of 41.39 indicates huge spread and variance in distribution.
- Kurtosis value of -0.62 indicates thin tail.
- Skewness value of 0.59 indicates Positive Skewness.
MDVP:Fhi(Hz):- Min-Max Range of 489.89 and Standard Deviation of 91.49 indicates huge spread and variance in distribution.
- Kurtosis value of 7.62 indicates thick tail.
- Skewness value of 2.542 indicates Positive Skewness in accordance with the thick tail denoted by Kurtosis.
MDVP:Flo(Hz):- Min-Max Range of 173.69 and Standard Deviation of 43.52 indicates huge spread and variance in distribution.
- Kurtosis value of 0.654 indicates presence of tail.
- Skewness value of 1.217 indicates Strong Positive Skewness with a long tail.
MDVP:Jitter(%):- Min-Max Range of 0.031480 and Standard Deviation of 0.004848 indicates wide spread and variance in distribution.
- Kurtosis value of 12.03 indicates very strong tail.
- Skewness value of 3.085 indicates Positive Skewness.
MDVP:Jitter(Abs):- Min-Max Range of 0.000253 and Standard Deviation of 0.000035 indicates wide spread and variance in distribution.
- Kurtosis value of 10.86 indicates very strong tail.
- Skewness value of 2.649 indicates Positive Skewness with data spread on the long tail.
MDVP:RAP:- Min-Max Range of 0.02076 and Standard Deviation of 0.00296 indicates fair spread of data and variance in distribution.
- Kurtosis value of 14.213 indicates very strong tail.
- Skewness value of 3.360 indicates Positive Skewness with data distributed among the long tail.
MDVP:PPQ:- Min-Max Range of 0.0186 and Standard Deviation of 0.00275 indicates wide spread and variance in distribution.
- Kurtosis value of 11.9639 indicates very strong tail.
- Skewness value of 3.0738 indicates Strong Positive Skewness with data spread on the long tail.
Jitter:DDP:- Min-Max Range of 0.0622 and Standard Deviation of 0.0089 indicates wide spread and variance in distribution.
- Kurtosis value of 14.2247 indicates very strong tail.
- Skewness value of 3.3620 indicates Strong Positive Skewness with data spread on the long tail.
MDVP:Shimmer:- Min-Max Range of 0.1095 and Standard Deviation of 0.0188 indicates wide spread and variance in distribution.
- Kurtosis value of 3.2383 indicates very strong tail.
- Skewness value of 1.6664 indicates Positive Skewness with data spread on the long tail.
MDVP:Shimmer(dB):- Min-Max Range of 1.2170 and Standard Deviation of 0.1948 indicates wide spread and variance in distribution.
- Kurtosis value of 5.1281 indicates strong tail.
- Skewness value of 1.9993 indicates Strong Positive Skewness with data spread on the tail.
Shimmer:APQ3:- Min-Max Range of 0.0519 and Standard Deviation of 0.0101 indicates wide spread and variance in distribution.
- Kurtosis value of 2.7201 indicates fair tail.
- Skewness value of 1.5805 indicates Positive Skewness with data spread on the tail.
Shimmer:APQ5:- Min-Max Range of 0.0737 and Standard Deviation of 0.0120 indicates wide spread and variance in distribution.
- Kurtosis value of 3.8742 indicates fair tail.
- Skewness value of 1.7986 indicates Positive Skewness with data spread on the tail.
MDVP:APQ:- Min-Max Range of 0.1305 and Standard Deviation of 0.0169 indicates wide spread and variance in distribution.
- Kurtosis value of 11.1632 indicates very strong tail.
- Skewness value of 2.6180 indicates Strong Positive Skewness with data spread on the long tail.
Shimmer:DDA:- Min-Max Range of 0.1557 and Standard Deviation of 0.0304 indicates wide spread and variance in distribution.
- Kurtosis value of 2.7206 indicates fair tail.
- Skewness value of 1.5806 indicates Positive Skewness with data spread on the tail.
NHR:- Min-Max Range of 0.3141 and Standard Deviation of 0.0404 indicates wide spread and variance in distribution.
- Kurtosis value of 21.9949 indicates very strong tail.
- Skewness value of 4.2207 indicates Strong Positive Skewness with data spread on the long tail.
HNR:- Min-Max Range of 24.606 and Standard Deviation of 4.4257 indicates wide spread and variance in distribution.
- Kurtosis value of 0.6160 indicates thin tail.
- Skewness value of -0.5143 indicates Slight Negative Skewness.
RPDE:- Min-Max Range of 0.4285 and Standard Deviation of 0.1039 indicates wide spread and variance in distribution.
- Kurtosis value of -0.9217 indicates thin tail.
- Skewness value of -0.1434 indicates Slight Negative Skewness.
DFA:- Min-Max Range of 0.2510 and Standard Deviation of 0.05533 indicates wide spread and variance in distribution.
- Kurtosis value of -0.68615 indicates thin tail.
- Skewness value of -0.03321 indicates Slight Negative Skewness.
spread1:- Min-Max Range of 5.5309 and Standard Deviation of 1.0902 indicates wide spread and variance in distribution.
- Kurtosis value of 0.05019 indicates thin tail.
- Skewness value of 0.4321 indicates Slight Positive Skewness.
spread2:- Min-Max Range of 0.4442 and Standard Deviation of 0.0834 indicates huge spread and variance in distribution.
- Kurtosis value of -0.08302 indicates thin tail.
- Skewness value of 0.1444 indicates Slight Positive Skewness.
D2:- Min-Max Range of 2.2478 and Standard Deviation of 0.3827 indicates huge spread and variance in distribution.
- Kurtosis value of 0.2203 indicates presence of tail.
- Skewness value of 0.4303 indicates Positive Skewness, but the Skewness is minimal.
PPE:- Min-Max Range of 0.48282 and Standard Deviation of 0.09011 indicates wide spread and variance in distribution.
- Kurtosis value of 0.5283 indicates presence of tail.
- Skewness value of 0.7974 indicates Positive Skewness.
Step 4: Exploratory Data Analysis (Univariate & Multivariate)¶
Dropping name from the dataframe.
In [10]:
parkinsons = parkinsons.drop(columns='name')
parkinsons.head()
Out[10]:
Univariate Analysis¶
In [11]:
#Segregating the Numeric Predictors for plotting
num_cols = ['MDVP:Fo(Hz)', 'MDVP:Fhi(Hz)', 'MDVP:Flo(Hz)', 'MDVP:Jitter(%)',
'MDVP:Jitter(Abs)', 'MDVP:RAP', 'MDVP:PPQ', 'Jitter:DDP',
'MDVP:Shimmer', 'MDVP:Shimmer(dB)', 'Shimmer:APQ3', 'Shimmer:APQ5',
'MDVP:APQ', 'Shimmer:DDA', 'NHR', 'HNR', 'RPDE', 'DFA',
'spread1', 'spread2', 'D2', 'PPE']
In [12]:
def plot_univariate_features(df):
"""
Helper function to plot Univariate features.
Input : Dataframe; Output : All Univariate plots for both Numeric and Categorical variables.
"""
print("Integer Columns = ",df.select_dtypes(include=['int32','int64']).columns)
print("Floating Point Columns = ",df.select_dtypes(include=['float64']).columns)
print("Object Columns = ",df.select_dtypes(include=['object']).columns)
print("Category Columns = ",df.select_dtypes(include=['category']).columns)
#sns.set_style(style='darkgrid')
int_cols = pd.Series(df.select_dtypes(include=['int32','int64']).columns)
for j in range(0,len(int_cols)):
f, axes = plt.subplots(1, 2, figsize=(10, 10))
sns.boxplot(df[int_cols[j]], ax = axes[0], palette='Greens_r')
sns.distplot(df[int_cols[j]], ax = axes[1], fit=norm)
plt.subplots_adjust(top = 1.5, right = 10, left = 8, bottom = 1)
float_cols = pd.Series(df.select_dtypes(include=['float64']).columns)
for j in range(0,len(float_cols)):
plt.Text('Figure for float64')
f, axes = plt.subplots(1, 2, figsize=(10, 10))
sns.boxplot(df[float_cols[j]], ax = axes[0], palette='Greens_r')
sns.distplot(df[float_cols[j]], ax = axes[1], fit=norm)
plt.subplots_adjust(top = 1.5, right = 10, left = 8, bottom = 1)
obj_cols = pd.Series(df.select_dtypes(include=['object']).columns)
for j in range(0,len(obj_cols)):
plt.subplots()
sns.countplot(df[obj_cols[j]])
In [13]:
plot_univariate_features(parkinsons[num_cols])
#f = pd.melt(parkinsons, value_vars=cols)
#g = sns.FacetGrid(f, col="variable", col_wrap=2, sharex=False, sharey=False, height=4, aspect=2)
#g = g.map(sns.distplot, "value")
List of features with Multiple Gausssian Mixtures: (Multimodal Distributions)
MDVP:Fhi(Hz)MDVP:RAPMDVP:PPQShimmer:APQ5HNRRPDEDFA
Shapiro Test for Normality check¶
In [14]:
normality_test = lambda x: shapiro(x.fillna(0))[1] < 0.01
normal = parkinsons[num_cols]
normal = normal.apply(normality_test)
print(~normal)
In [15]:
non_normal_cols = ['MDVP:Fo(Hz)','MDVP:Fhi(Hz)','MDVP:Flo(Hz)','MDVP:Jitter(%)',
'MDVP:Jitter(Abs)','MDVP:RAP','MDVP:PPQ','Jitter:DDP','MDVP:Shimmer',
'MDVP:Shimmer(dB)','Shimmer:APQ3','Shimmer:APQ5','MDVP:APQ','Shimmer:DDA','NHR','HNR','RPDE','PPE']
non_normal_cols
Out[15]:
Skewness Check¶
In [16]:
plt.figure(figsize=(10,10))
parkinsons[num_cols].skew().sort_values().plot(kind='barh')
plt.show()
Outliers Detection and Treatment¶
In [17]:
def outliers_IQR(df):
"""
Helper function to detect Outliers in the Dataframe
Input : Dataframe; Output : Dataframe containing Percentage and Number of Outliers.
"""
columns = df.columns
outliers_list = []
no_of_outliers = []
for c in df.columns:
Q1 = df[c].quantile(0.25)
Q3 = df[c].quantile(0.75)
IQR = Q3 - Q1
df_outliers = np.where((df[c] < (Q1 - 1.5 * IQR)) | (df[c] > (Q3 + 1.5 * IQR)))
no_of_outliers.append(len(df_outliers[0]))
outliers_list.append(round((len(df_outliers[0]) / len(df[c]) * 100),2))
print("\n")
outliers_df = pd.DataFrame({"Percentage_of_Outliers":outliers_list, "No_of_Outliers":no_of_outliers}, index=df.columns)
return outliers_df.sort_values(by="Percentage_of_Outliers", ascending=False)
In [18]:
outliers_IQR(parkinsons[non_normal_cols])
Out[18]:
Imputing the Outliers
In [19]:
# Removing outliers (data points beyond 1.5xIQR)
for col in non_normal_cols:
Q1 = parkinsons[col].quantile(0.25)
Q3 = parkinsons[col].quantile(0.75)
IQR = Q3 - Q1
lower_lim = Q1 - 1.5*IQR
upper_lim = Q3 + 1.5*IQR
parkinsons.loc[(parkinsons[col] > upper_lim), col] = np.nan
parkinsons.loc[(parkinsons[col] < lower_lim), col] = np.nan
print('Outliers are set to NaN')
print('Number of rows impacted due to Outlier treatment : ', parkinsons.isnull().any().sum())
In [20]:
#Imputing the Outlier NaNs with Median values
parkinsons[non_normal_cols] = parkinsons[non_normal_cols].apply(lambda x: x.fillna(x.median()),axis=0)
parkinsons.sample(3)
Out[20]:
In [21]:
parkinsons.isnull().any().sum() #Sanity check post imputation
Out[21]:
Visualising the Target variable status¶
In [22]:
parkinsons['status'].value_counts()
Out[22]:
In [23]:
sns.countplot(y='status', data=parkinsons);
In [24]:
colors = ["lightgreen", "silver"]
labels = ['Yes', 'No']
plt.pie(parkinsons['status'].value_counts(), labels = labels, colors = colors, autopct='%1.1f%%', shadow=True)
plt.title("Distribution of Parkinsons patients in the dataset")
plt.show()
Around 3/4th of the records are diagnosed with Parkinsons. This is mainly because every patient has 6 records in the given dataset, which is inducing 'Sample Bias'.
However, since the objective of this dataset is to identify Parkinson's in early stages, we are interested in the True Positive Rate more than the Negative Records (which will not be as costly as misclassifying a Parkinson's patient as healthy).
Hence, we will not do any resampling for the Class Imbalance and will be looking at additional metrics such as ROC AUC, F1-Score, Recall etc in addition to the Accuracy.
Multivariate Analysis¶
In [25]:
def plot_predictor_target_features(df):
"""
Helper function to plot multivariate features.
Input : Dataframe; Output : One-to-Many Multivariate plots for Predictors vs Target feature.
"""
print("All Columns = ",df.columns)
predictor_cols = pd.Series(df.columns)
for j in range(0,len(predictor_cols)):
f, axes = plt.subplots(1, 2, figsize=(10, 10))
sns.boxplot(y=df[predictor_cols[j]], x=parkinsons['status'], ax = axes[0], palette='Accent')
sns.swarmplot(y=df[predictor_cols[j]], x=parkinsons['status'], ax = axes[0], palette='Dark2')
sns.violinplot(y=df[predictor_cols[j]], x=parkinsons['status'], ax = axes[1], palette='YlGn')
plt.subplots_adjust(top = 1.5, right = 10, left = 8, bottom = 1)
In [26]:
plot_predictor_target_features(parkinsons[num_cols])
Pairplot of all variables in the dataset¶
In [27]:
plt.figure(figsize=(20,20))
sns.pairplot(parkinsons, hue='status', diag_kind = 'kde')
plt.show()
Correlation Matrix¶
In [28]:
plt.figure(figsize=(14,14), dpi=300)
sns.heatmap(parkinsons.corr(), annot=True, linewidths = 0.3, fmt = '0.2f', cmap = 'YlGnBu', square=True)
plt.show()
Highly correlated features with target variable status¶
In [29]:
plt.figure(figsize=(15,15))
parkinsons.corr()[['status']].sort_values(by='status', ascending=True).plot(kind='barh')
plt.show()
Highly correlated Independent variables¶
In [30]:
#Identifying highly correlated Independent variables (Predictors)
high_corr_var = parkinsons.corr().abs().unstack() #abs() considers absolute values of correlation coefficients.
high_corr_var = high_corr_var.sort_values(kind = "quicksort", ascending = False)
print('No of highly correlated instances : ',len(high_corr_var[(high_corr_var > 0.8) & (high_corr_var < 1)]))
print("\n")
print("Percentage of multicollinear variables : ", (len(high_corr_var[(high_corr_var > 0.8) & (high_corr_var < 1)]) / len(parkinsons) * 100))
print("\n")
print(high_corr_var[(high_corr_var > 0.8) & (high_corr_var < 1)])
Observations from the Multivariate Analysis & Correlation Matrix:¶
MDVP:Fhi(Hz),DFA,MDVP:Fo(Hz),MDVP:Flo(Hz),RPDE,spread2,D2have weak correlation with other predictors.PPE,spread1,MDVP:APQ,MDVP:Shimmer,MDVP:Shimmer(dB),Shimmer:APQ5,MDVP:PPQ,Jitter:DDP,MDVP:RAP,MDVP:Jitter(Abs),Shimmer:APQ3,MDVP:Jitter(%),Shimmer:DDA,NHRexhibit multicollinearity, which will be further checked by Variance Inflation Factor (VIF) in the next section.HNRexhibits negative correlation with the target variablestatus.
Detecting Multicollinear features using Variance Inflation Factor (VIF)¶
In [31]:
def calculate_vif(X):
vif_features = pd.DataFrame()
vif_features["Features"] = X.columns
vif_features["VIF Score"] = [vif(X.values, i) for i in range(X.shape[1])]
return(vif_features)
In [32]:
vif_df = calculate_vif(parkinsons[num_cols])
vif_df.sort_values(by='VIF Score', ascending=False)
Out[32]:
In [33]:
vif_df[vif_df['VIF Score']<10].sort_values(by='VIF Score', ascending=False)
Out[33]:
Step 5: Split the data into Train and Test set¶
Separating the Predictors and Target variables in X & y¶
In [34]:
X = parkinsons[num_cols]
y = parkinsons['status']
Split the Train and Test set¶
In [35]:
#Stratify is used to preserve proportion of target classes in splits, since the Class is Imbalanced
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.3, random_state = 24)
View the shapes of Train and Test sets¶
In [36]:
display(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
Step 6: Data Preprocessing¶
Scaling the data using RobustScaler¶
In [37]:
#Outliers in our dataset will be better handled by RobustScaler()
scaler = RobustScaler()
#Fitting the Training set
scaler.fit(X_train)
Out[37]:
Note: Fit using X_train and transform both X_train and X_test. It is better to construct Pipelines for such Preprocessing steps, though it is out of scope of this current assignment.
In [38]:
scaler.transform(X_train)
scaler.transform(X_test)
Out[38]:
Note:
- RobustScaler() is used instead of StandardScaler() or MinMaxScaler() since RobustScaler can handle Outliers present in the data to a better extent.
- Scaling and Centering the data is a must for Gaussian algorithms such as 'Support Vector Machines' with Radial Basis Kernel and distance-based algorithms such as K-Nearest Neighbors.
- However, the scaling is not a must for Non-parametric methods such as Trees.
Step 7: Model Building - Training Classifiers¶
In [39]:
def classifier_metrics(classifier):
"""
Helper function to plot Univariate features.
Input : Classifier;
Output : Accuracy, Classification Report, Confusion Matrix, ROC AUC Curve, Precision-Recall Curve
"""
y_true, y_pred = y_test, classifier.predict(X_test)
train_accuracy = classifier.score(X_train, y_train).round(3)
test_accuracy = classifier.score(X_test, y_test).round(3)
print("Train Accuracy : ", train_accuracy)
print("Test Accuracy : ", test_accuracy)
# Classification Report
print("\nClassification Report")
print('\n{}'.format(classification_report(y_true, y_pred)))
# Precision Score
clf_precision = precision_score(y_true, y_pred)
print('\nPrecision Score:\n', clf_precision.round(3))
# Recall Score
clf_recall = recall_score(y_true, y_pred)
print('\nRecall Score:\n', clf_recall.round(3))
# F1 Score
clf_f1 = f1_score(y_true, y_pred)
print('\nF1 Score:\n', clf_f1.round(3))
# ROC AUC Score
clf_roc_auc = roc_auc_score(y_true, y_pred)
print('\nROC AUC Score:\n', clf_roc_auc.round(3))
#Gini-Coefficient
clf_gini_coeff = 2 * clf_roc_auc -1
print("\nGini Coefficient :\n", clf_gini_coeff.round(2))
# Confusion Matrix
clf_conf_matrix = confusion_matrix(y_true, y_pred)
print('\nConfusion Matrix:\n')
sns.heatmap(clf_conf_matrix, annot=True, square=True, fmt='g')
#ROC AUC Curve
clf_fpr, clf_tpr, clf_thresholds = roc_curve(y_true, classifier.predict_proba(X_test)[:,1])
plt.figure(figsize = (14 , 6))
plt.plot(clf_fpr, clf_tpr, label = 'Classifier AUC (area = {})'.format(clf_roc_auc.round(2)))
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend(loc = 'lower right')
plt.show()
#Precision-Recall Curve
plt.figure(figsize=(14,6))
plot_precision_recall_curve(classifier, X_test, y_test)
plt.title("Precision-Recall AUC Curve")
plt.legend(loc = 'lower right')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.show()
return train_accuracy, test_accuracy, clf_conf_matrix, clf_precision, clf_recall, clf_f1, clf_roc_auc, clf_gini_coeff,clf_fpr, clf_tpr, clf_thresholds
Support Vector Machines¶
SVM with Polynomial Kernel
In [40]:
svm_clf = SVC(kernel = 'poly', C = 100, gamma='auto', probability=True)
In [41]:
svm_clf.fit(X_train, y_train)
Out[41]:
In [42]:
svm_train_accuracy, svm_test_accuracy, svm_clf_conf_matrix, svm_clf_precision, svm_clf_recall, svm_clf_f1, svm_clf_roc_auc, svm_clf_gini_coeff, svm_clf_fpr, svm_clf_tpr, svm_clf_thresholds = classifier_metrics(svm_clf)
SVM with Radial Basis Kernel
In [43]:
svm_clf_rbf = SVC(kernel = 'rbf', C = 100, gamma='auto', probability=True)
In [44]:
svm_clf_rbf.fit(X_train, y_train)
Out[44]:
In [45]:
svm_train_accuracy, svm_test_accuracy, svm_clf_conf_matrix, svm_clf_precision, svm_clf_recall, svm_clf_f1, svm_clf_roc_auc, svm_clf_gini_coeff, svm_clf_fpr, svm_clf_tpr, svm_clf_thresholds = classifier_metrics(svm_clf_rbf)
Logistic Regression¶
In [46]:
log_reg_clf = LogisticRegression(solver='lbfgs', n_jobs=-1)
In [47]:
log_reg_clf.fit(X_train, y_train)
Out[47]:
In [48]:
lr_train_accuracy, lr_test_accuracy, lr_clf_conf_matrix, lr_clf_precision, lr_clf_recall, lr_clf_f1, lr_clf_roc_auc, lr_clf_gini_coeff, lr_clf_fpr, lr_clf_tpr, lr_clf_thresholds = classifier_metrics(log_reg_clf)
Logistic Regression, being a Parametric Method has the closed Train and Test Accuracy Scores, showing that the model hasn't overfitted.
Naive Bayes Classifier¶
In [49]:
gnb_clf = GaussianNB()
In [50]:
gnb_clf.fit(X_train, y_train)
Out[50]:
In [51]:
gnb_train_accuracy, gnb_test_accuracy, gnb_clf_conf_matrix, gnb_clf_precision, gnb_clf_recall, gnb_clf_f1, gnb_clf_roc_auc, gnb_clf_gini_coeff, gnb_clf_fpr, gnb_clf_tpr, gnb_clf_thresholds = classifier_metrics(gnb_clf)
Given dataset has lot of multicollinear features, which defeats the core 'Naive' assumption of Gaussian Naive Bayes algorithm. This is visible from the very-average results displayed by the model.
Decision Tree Classifier¶
In [52]:
dt_clf = DecisionTreeClassifier(criterion='entropy', random_state=24)
In [53]:
dt_clf.fit(X_train, y_train)
Out[53]:
In [54]:
dt_train_accuracy, dt_test_accuracy, dt_clf_conf_matrix, dt_clf_precision, dt_clf_recall, dt_clf_f1, dt_clf_roc_auc, dt_clf_gini_coeff, dt_clf_fpr, dt_clf_tpr, dt_clf_thresholds = classifier_metrics(dt_clf)
Visualising the Decision Tree¶
In [55]:
text_representation = tree.export_text(dt_clf)
print(text_representation)
In [56]:
import graphviz
dot_data = tree.export_graphviz(dt_clf, out_file=None,
feature_names=X.columns,
class_names=['No', 'Yes'],
filled=True)
graph = graphviz.Source(dot_data, format="png")
graph
Out[56]:
In [57]:
! pip install dtreeviz
In [57]:
from dtreeviz.trees import dtreeviz # remember to load the package
viz = dtreeviz(dt_clf, X, y,
target_name="Status",
feature_names=X.columns,
class_names=['No', 'Yes'])
viz
Out[57]:
Step 8: Model Building - Meta-classifier¶
Stacking Classifier using SVM, LogReg, NB and Decision Tree Models¶
In [58]:
stk_clf = StackingClassifier(estimators=[('SVC', svm_clf_rbf), ('LogR', log_reg_clf), ('GNB', gnb_clf), ('DT', dt_clf)],
final_estimator=dt_clf)
#DecisionTreeClassifier() is used as the final_estimator for this StackingClassifier.
In [59]:
stk_clf.fit(X_train, y_train)
Out[59]:
In [60]:
stk_train_accuracy, stk_test_accuracy, stk_clf_conf_matrix, stk_clf_precision, stk_clf_recall, stk_clf_f1, stk_clf_roc_auc, stk_clf_gini_coeff, stk_clf_fpr, stk_clf_tpr, stk_clf_thresholds = classifier_metrics(stk_clf)
- For this particular dataset, StackingClassifier doesn't provide better results than the standalone models such as SVM and Log-Reg but still the results are not at par with Decision Tree Classifier.
Step 9: Model Building - Ensembles¶
Let us start with a Bagging Ensemble - Random Forest!
Random Forest Classifier¶
In [61]:
rf_clf = RandomForestClassifier(n_estimators=100, criterion='entropy', n_jobs=-1, oob_score=True, random_state=24)
In [62]:
rf_clf.fit(X_train, y_train)
Out[62]:
In [63]:
print("Out-of-bag Evaluation Score for Random Forest : ", rf_clf.oob_score_.round(3))
In [64]:
rf_train_accuracy, rf_test_accuracy, rf_clf_conf_matrix, rf_clf_precision, rf_clf_recall, rf_clf_f1, rf_clf_roc_auc, rf_clf_gini_coeff, rf_clf_fpr, rf_clf_tpr, rf_clf_thresholds = classifier_metrics(rf_clf)
In [65]:
def plot_feature_rank(clf):
feature_rank = pd.DataFrame({
'feature' : X_train.columns,
'importance' : clf.feature_importances_.round(3)
}
)
feature_rank = feature_rank.sort_values('importance', ascending=False)
print("Amount of variance explained by the predictors\n")
feature_rank['cumulative_sum'] = feature_rank['importance'].cumsum()*100
plt.figure(figsize=(8,6))
sns.barplot(y='feature', x='importance', data=feature_rank)
plt.show()
return feature_rank
In [66]:
rf_feature_rank = plot_feature_rank(rf_clf)
rf_feature_rank
Out[66]:
In [67]:
sns.lineplot(y=rf_feature_rank['cumulative_sum'], x=np.arange(0,22,1))
Out[67]:
- We can see that the
PPEfeature is the most important feature as per RandomForestClassifier and DecisionTreeClassifier algorithms. HNRwhich has negative correlation with target variable is labeled as 2nd most important feature by DecisionTreeClassifier however, it is given moderate priority by RandomForestClassifier.
Hyperparameter Tuning for Random Forest Classifier¶
In [68]:
rf_clf_1 = RandomForestClassifier(random_state=24, n_jobs=-1, oob_score=True)
rf_clf_1.fit(X_train, y_train)
Out[68]:
In [69]:
param_grid = {
'n_estimators':[1,50,100,150,200],
'max_depth':[10,20,50,80,100]
}
rf_tuned_clf = GridSearchCV(rf_clf_1,param_grid=param_grid, scoring='roc_auc',cv=StratifiedKFold(n_splits=10),n_jobs=-1)
In [70]:
rf_tuned_clf.fit(X_train,y_train)
Out[70]:
In [71]:
rf_tuned_clf.best_estimator_
Out[71]:
In [72]:
print("Out-of-bag Evaluation Score for Random Forest Tuned : ", rf_tuned_clf.best_estimator_.oob_score_.round(3))
In [73]:
rf_tuned_clf.best_params_
Out[73]:
In [74]:
rf_tuned_cv = pd.DataFrame(rf_tuned_clf.cv_results_)
rf_tuned_cv
Out[74]:
In [75]:
rf_tuned_train_accuracy, rf_tuned_test_accuracy, rf_tuned_clf_conf_matrix, rf_tuned_clf_precision, rf_tuned_clf_recall, rf_tuned_clf_f1, rf_tuned_clf_roc_auc, rf_tuned_clf_gini_coeff, rf_tuned_clf_fpr, rf_tuned_clf_tpr, rf_tuned_clf_thresholds = classifier_metrics(rf_tuned_clf)
Feature Importance for Random Forest Classifier Tuned¶
In [76]:
rf_tuned_feature_rank = plot_feature_rank(rf_tuned_clf.best_estimator_)
rf_tuned_feature_rank
Out[76]:
In [77]:
sns.lineplot(y=rf_tuned_feature_rank['cumulative_sum'], x=np.arange(0,22,1))
Out[77]:
So, RandomForest has provided spectacular results, as expected. Let us try with a Boosting Ensemble - AdaBoost (Adaptive Boosting)
AdaBoost Classifier¶
In [78]:
ada_clf = AdaBoostClassifier(algorithm='SAMME.R', random_state=24)
#Default Estimator is DecisionTreeClassifier() with max_depth = 1 (Tree Stump)
In [79]:
ada_clf.fit(X_train, y_train)
Out[79]:
In [80]:
ada_train_accuracy, ada_test_accuracy, ada_clf_conf_matrix, ada_clf_precision, ada_clf_recall, ada_clf_f1, ada_clf_roc_auc, ada_clf_gini_coeff, ada_clf_fpr, ada_clf_tpr, ada_clf_thresholds = classifier_metrics(ada_clf)
Hyperparameter Tuning for AdaBoost Classifier with different Base Estimators¶
SVM, Logistic Regression and Random Forest Classifiers are provided as the different Base Estimators for the AdaBoost Classifier. Let us see which classifier is picked by the GridSearch.
In [81]:
param_grid = {
'base_estimator' : [svm_clf_rbf, log_reg_clf, rf_clf],
'n_estimators': [1,50,100,150,200],
'learning_rate': [0.01, 0.05, 0.1, 1.0]
}
ada_tuned_clf = GridSearchCV(AdaBoostClassifier(random_state=24),param_grid=param_grid, scoring='roc_auc',cv=StratifiedKFold(n_splits=10),n_jobs=-1)
In [82]:
ada_tuned_clf.fit(X_train, y_train)
Out[82]:
In [83]:
ada_tuned_clf.best_params_
Out[83]:
- We can see that out of the given Classifiers, RandomForest Classifier is picked by the GridSearch based on the metrics.
- Note that we have used a Bagging Ensemble 'RandomForest' as a Base Estimator for a Boosting Ensemble 'AdaBoost'!
In [84]:
ada_tuned_train_accuracy, ada_tuned_test_accuracy, ada_tuned_clf_conf_matrix, ada_tuned_clf_precision, ada_tuned_clf_recall, ada_tuned_clf_f1, ada_tuned_clf_roc_auc, ada_tuned_clf_gini_coeff, ada_tuned_clf_fpr, ada_tuned_clf_tpr, ada_tuned_clf_thresholds = classifier_metrics(ada_tuned_clf)
So, using a Bagging Ensemble 'RandomForest' as a Base Estimator for a Boosting Ensemble 'AdaBoost' has yielded the BEST results so far out of all the models.
Let us try with the Gradient Boosting methods next.
XGBoost Classifier¶
Instantiating the XGBoost (Extreme Gradient Boosting) Classifier. We have used 'gbtree' which used Decision Trees as the base estimator. Also the objective is 'binary:logistic' which uses Cross-Entropy Log Loss for arriving at the classification predictions.
In [85]:
xgb_clf = xgb.XGBClassifier(
booster='gbtree',
objective='binary:logistic',
random_state=24,
n_jobs=-1)
Now, let us fit the XGB model. Note, that we have used early_stopping_rounds = 10, which means that the model will NOT grow the entire tree and prune it back (unlike Random Forest) and the tree growth will stop if there is no improvement in the eval_metric for 10 rounds (epochs).
In [86]:
xgb_clf.fit(
X_train,
y_train,
verbose=True,
early_stopping_rounds=10, #Stop the Tree Growth when there is no visible improvement for 10 iterations
eval_metric='auc', #ROC AUC Curve
eval_set=[(X_test, y_test)]
)
Out[86]:
In [87]:
xgb_clf.best_score
Out[87]:
In [88]:
xgb_train_accuracy, xgb_test_accuracy, xgb_clf_conf_matrix, xgb_clf_precision, xgb_clf_recall, xgb_clf_f1, xgb_clf_roc_auc, xgb_clf_gini_coeff, xgb_clf_fpr, xgb_clf_tpr, xgb_clf_thresholds = classifier_metrics(xgb_clf)
Hyperparameter Tuning for XGBoost¶
In [89]:
xgb_estimator = xgb.XGBClassifier(
objective='binary:logistic',
seed=24,
random_state=24
)
In [90]:
param_grid = {
'max_depth' : [3,4,5,6], #Tree Depth
'learning_rate' : [0.1,0.01,0.05], #Shrinkage Parameter to avoid being Overfit
'gamma' : [0,0.25,1.0], #Regularization Parameter
'reg_lambda' : [0,1.0,10.0], #Regularization Parameter
'scale_pos_weight' : [1,3] #For Class Imbalance handling
}
xgb_tuned_clf = GridSearchCV(xgb_estimator,param_grid=param_grid, scoring='roc_auc',cv=StratifiedKFold(n_splits=10),n_jobs=-1)
In [91]:
xgb_tuned_clf.fit(
X_train,
y_train,
verbose=True,
early_stopping_rounds=10, #Stop the Tree Growth when there is no visible improvement for 10 iterations
eval_metric='auc', #ROC AUC Score
eval_set=[(X_test, y_test)]
)
Out[91]:
In [92]:
xgb_tuned_clf.best_params_
Out[92]:
In [93]:
xgb_tuned_clf.best_estimator_
Out[93]:
In [94]:
xgb_tuned_clf.best_score_
Out[94]:
In [95]:
xgb_tuned_train_accuracy, xgb_tuned_test_accuracy, xgb_tuned_clf_conf_matrix, xgb_tuned_clf_precision, xgb_tuned_clf_recall, xgb_tuned_clf_f1, xgb_tuned_clf_roc_auc, xgb_tuned_clf_gini_coeff, xgb_tuned_clf_fpr, xgb_tuned_clf_tpr, xgb_tuned_clf_thresholds = classifier_metrics(xgb_tuned_clf)
Visualizing the XGBoost Tree with Lone Estimator¶
In [96]:
xgb_lone_estimator = xgb.XGBClassifier(
objective='binary:logistic',
seed=24,
subsample=0.9,
colsample_bytree=0.5,
gamma = 0.25,
learning_rate=0.05,
max_depth=4,
reg_lambda=10.0,
scale_pos_weight=3,
n_estimator=1 #Single Tree for Visualisation purpose
)
In [97]:
xgb_lone_estimator.fit(X_train, y_train)
Out[97]:
In [98]:
xgb_bst = xgb_lone_estimator.get_booster()
for importance_type in ('weight', 'gain', 'cover', 'total_gain', 'total_cover'):
print('%s: ' % importance_type, xgb_bst.get_score(importance_type=importance_type))
node_params = {
'shape' : 'box',
'style' : 'filled, rounded',
'fillcolor' : '#78cbe'
}
leaf_params = {
'shape' : 'box',
'style' : 'filled',
'fillcolor' : '#e48038'
}
In [99]:
xgb.to_graphviz(xgb_lone_estimator,
num_trees=0,
size="5,5",
condition_node_params=node_params,
leaf_node_params=leaf_params
)
Out[99]:
In [100]:
xgb_feature_rank = plot_feature_rank(xgb_tuned_clf.best_estimator_)
xgb_feature_rank
Out[100]:
In [101]:
sns.lineplot(y=xgb_feature_rank['cumulative_sum'], x=np.arange(0,22,1))
Out[101]:
LightGBM Classifier¶
In [102]:
lgbm_clf = LGBMClassifier(
boosting_type='gbdt', #Gradient Boosting Decision Trees
objective='binary',
n_estimators=100,
learning_rate=0.5,
is_unbalanced=True,
n_jobs=-1,
random_state=24
)
In [103]:
lgbm_clf.fit(X_train, y_train)
Out[103]:
In [104]:
lgbm_train_accuracy, lgbm_test_accuracy, lgbm_clf_conf_matrix, lgbm_clf_precision, lgbm_clf_recall, lgbm_clf_f1, lgbm_clf_roc_auc, lgbm_clf_gini_coeff, lgbm_clf_fpr, lgbm_clf_tpr, lgbm_clf_thresholds = classifier_metrics(lgbm_clf)
Now, let us try out (greedy) Stacking with Random Forest, XGBoost and LightGBM models!
In [105]:
greedy_stk_clf = StackingClassifier(estimators=[('RF', rf_tuned_clf), ('XBG', xgb_tuned_clf), ('LGBM', lgbm_clf)], final_estimator=xgb_tuned_clf)
In [106]:
greedy_stk_clf.fit(X_train, y_train)
Out[106]:
In [107]:
greedy_stk_train_accuracy, greedy_stk_test_accuracy, greedy_stk_clf_conf_matrix, greedy_stk_clf_precision, greedy_stk_clf_recall, greedy_stk_clf_f1, greedy_stk_clf_roc_auc, greedy_stk_clf_gini_coeff, greedy_stk_clf_fpr, greedy_stk_clf_tpr, greedy_stk_clf_thresholds = classifier_metrics(greedy_stk_clf)
Step 10: Model Comparison¶
List of Model Evaluation Metrics:
- Accuracy (1 - Misclassification Rate)
- Precision (TP/(TP+FP))
- Recall (TP/(TP+FN))
- F1-Score (Harmonic Mean of Precision and Recall)
- ROC AUC Score (Area under the ROC Curve - FPR vs TPR)
- Gini Coefficient (2xROCAUC - 1)
Note:
We will look at all the parameters due to the inherent Class Imbalance in the given dataset. However, since the model is going to be used as a preliminary investigation step for a Clinic Appointment, we are more interested in reducing the 'False Negatives' and hence 'Recall' becomes one of the first metrics to be evaluated in relation to other metrics.
In [108]:
metrics = {'Classifier':['Logistic Regression',
'Gaussian Naive Bayes',
'Support Vector Machine',
'Decision Trees',
'Random Forest',
'RF Hyperparameter Tuned',
'AdaBoost',
'AdaBoost Hyperparameter Tuned',
'Stacking Classifier',
'XGBoost',
'XGB Hyperparameter Tuned',
'LightGBM'],
'Accuracy' : [lr_test_accuracy,
gnb_test_accuracy,
svm_test_accuracy,
dt_test_accuracy,
rf_test_accuracy,
rf_tuned_test_accuracy,
ada_test_accuracy,
ada_tuned_test_accuracy,
stk_test_accuracy,
xgb_test_accuracy,
xgb_tuned_test_accuracy,
lgbm_test_accuracy],
'Precision':[lr_clf_precision,
gnb_clf_precision,
svm_clf_precision,
dt_clf_precision,
rf_clf_precision,
rf_tuned_clf_precision,
ada_clf_precision,
ada_tuned_clf_precision,
stk_clf_precision,
xgb_clf_precision,
xgb_tuned_clf_precision,
lgbm_clf_precision],
'Recall':[lr_clf_recall,
gnb_clf_recall,
svm_clf_recall,
dt_clf_recall,
rf_clf_recall,
rf_tuned_clf_recall,
ada_clf_recall,
ada_tuned_clf_recall,
stk_clf_recall,
xgb_clf_recall,
xgb_tuned_clf_recall,
lgbm_clf_recall],
'F1-Score':[lr_clf_f1,
gnb_clf_f1,
svm_clf_f1,
dt_clf_f1,
rf_clf_f1,
rf_tuned_clf_f1,
ada_clf_f1,
ada_tuned_clf_f1,
stk_clf_f1,
xgb_clf_f1,
xgb_tuned_clf_f1,
lgbm_clf_f1],
'ROC AUC':[lr_clf_roc_auc,
gnb_clf_roc_auc,
svm_clf_roc_auc,
dt_clf_roc_auc,
rf_clf_roc_auc,
rf_tuned_clf_roc_auc,
ada_clf_roc_auc,
ada_tuned_clf_roc_auc,
stk_clf_roc_auc,
xgb_clf_roc_auc,
xgb_tuned_clf_roc_auc,
lgbm_clf_roc_auc],
'Gini Coeff':[lr_clf_gini_coeff,
gnb_clf_gini_coeff,
svm_clf_gini_coeff,
dt_clf_gini_coeff,
rf_clf_gini_coeff,
rf_tuned_clf_gini_coeff,
ada_clf_gini_coeff,
ada_tuned_clf_gini_coeff,
stk_clf_gini_coeff,
xgb_clf_gini_coeff,
xgb_tuned_clf_gini_coeff,
lgbm_clf_gini_coeff]
}
model_eval_metrics = pd.DataFrame(metrics)
model_eval_metrics = model_eval_metrics.set_index('Classifier')
In [109]:
model_eval_metrics
Out[109]:
In [111]:
plt.figure(figsize = (14 , 8))
plt.plot(lr_clf_fpr, lr_clf_tpr, label = 'Logistic Regression (area = {})'.format(lr_clf_roc_auc.round(2)))
plt.plot(gnb_clf_fpr, gnb_clf_tpr, label = 'Naive Bayes (area = {})'.format(gnb_clf_roc_auc.round(2)))
plt.plot(svm_clf_fpr, svm_clf_tpr, label = 'SVM (area = {})'.format(svm_clf_roc_auc.round(2)))
plt.plot(dt_clf_fpr, dt_clf_tpr, label = 'Decision Tree Classifier (area = {})'.format(dt_clf_roc_auc.round(2)))
plt.plot(rf_tuned_clf_fpr, rf_tuned_clf_tpr, label = 'Random Forest Tuned (area = {})'.format(rf_tuned_clf_roc_auc.round(2)))
plt.plot(ada_tuned_clf_fpr, ada_tuned_clf_tpr, label = 'AdaBoost Tuned (area = {})'.format(ada_tuned_clf_roc_auc.round(2)))
plt.plot(xgb_tuned_clf_fpr, xgb_tuned_clf_tpr, label = 'XGBoost Tuned (area = {})'.format(xgb_tuned_clf_roc_auc.round(2)))
plt.plot(lgbm_clf_fpr, lgbm_clf_tpr, label = 'LightGBM (area = {})'.format(lgbm_clf_roc_auc.round(2)))
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC AUC - Classifiers Comparison')
plt.legend(loc = 'lower right')
plt.show()
In [112]:
print("Best Performance Model :\n")
model_eval_metrics.idxmax()
Out[112]:
Summary¶
In [113]:
plt.figure(figsize=(12,12))
sns.heatmap(model_eval_metrics, annot=True, linewidths = 0.3, fmt='.2f', cmap='YlGnBu').set_title('Classifier Dashboard')
plt.show()
Given dataset is trained and fitted with the following Classifiers:
Standalone Classifiers:
- Logistic Regression
- Gaussian Naive Bayes
- Support Vector Machines
- Decision Tree Classifier
Ensembles:
Meta-Classifier with Logistic Regression, Gaussian Naive Bayes and Support Vector Machines stacked up.
Random Forest Classifier (Bagging Ensemble)
- AdaBoost Classifier (Boosting Ensemble with RandomForest as the base estimator)
- XGBoost Classifier (Boosting Ensemble)
- LightGBM Classifier (Boosting Ensemble)
Finally, all the classifier metrics are compared and evaluated using:
- ROC AUC Score
- Precision-Recall AUC Score
- Test Accuracy Score
- F1-Score
Inference from Classifier Comparison:
- Given dataset has Class Imbalance with classes skewed towards the positive class - mainly because for each patient a total of 6 records are available. This may cause 'Sample Bias' in the model inherently.
- Given dataset has lot of multicollinear features with inherent dependencies. This will impact 'Parametric Methods' such as Logistic Regression and SVM, in the absence of regularisation.
- It is observed that both Log-Reg and SVM (default values) have given average results. However, SVM with high regularisation (C=100) has provided a Recall Score of 1.00, though the accuracy score is not the best among the classifiers.
- Gaussian Naive Bayes is one of the worst performing Classifiers for this dataset mainly because of the inherent dependencies among the predictors, which is against the 'Naive' assumption of Naive Bayes that the predictors are independent.
- Decision Tree Classifier is able to provide better generalised results when compared to the other classifiers, this is mainly because of the 'Non-Parametric' nature of the algorithm.
- In contrast to the standalone Classifiers, it is observed that the Ensembles are able to provide better results with this dataset.
- Random Forest Classifier, AdaBoost (with Random Forest as Base Estimator), XGBoost and LightGBM have performed excellently in terms of the overall ROC AUC Score and Accuracy. Feature Importance is also plotted for all the tree-based models for model interpretability.
- From the ROC AUC Curve for all the classifiers, we can infer that,
- If our objective is to arrest False Positive Rate, then 'XGBoost' is the preferred model, considering it is the first model to reach a True Positive Rate of 1.00 for a False Negative Rate of 0.30.
- If our objective is to improve True Positive Rate at the expense of few False Positives, 'AdaBoost' and 'Random Forest' classifiers are preferrable as they provide overall higher True Positive Rate score.
- In addition, Tree-based models have the advantage to get Feature Importance, and also can interpret the model using 'Partial Dependence Plots', 'Waterfall Charts' etc.
- If True Positive Rate is the only objective at the expense of Model Interpretability, we can go for extremely regularized 'black-box' models such as 'SVM' (Radial Basis Kernel at C=100) has provided a Recall score of 1.0 on the test dataset. However, it would be challenging to explain the model interpretability to stakeholders.
Since the objective of this entire exercise is to build a model that will predict Parkinsons's which will be subsequently used as a preliminary alert check for the patient to book a doctor's appointment, False Positives will not have much impact on the results (doctor's appointment will resolve it in next steps), However, we are more interested in reducing the False Negatives, therefore we go for models that provide higher True Positive Rate (Recall score = TP/(TP+FN)). Hence, 'AdaBoost with Random Forest as Base Estimator' and 'XGBoost' models are recommended here with the thresholds set using ROC AUC & Precision-Recall Curve.